Skip to content

Ignore heading anchor ids in class/id rules - #19

Merged
Alaa-abdulridha merged 2 commits into
mainfrom
heading-anchor-names
Nov 4, 2026
Merged

Alaa-abdulridha merged 2 commits into
mainfrom
heading-anchor-names

Conversation

@Alaa-abdulridha

@Alaa-abdulridha Alaa-abdulridha commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Sites give headings ids made from their own text, so tables of contents can link to them: WordPress uses h- plus the slug (h-what-is-social-shopping), other sites use the plain slug, sometimes with a counter (travel-related-coverage-2). Documentation generators put the same kind of id on the section a heading opens (Sphinx <section id="cookies">, Django <section id="s-...">).

The class/id rules read these ids as the names of sections. "social" in what-is-social-media-marketing, "sponsor" in h-sponsorship-deals and "related" in travel-related-coverage matched Boilerplate's noise names, "comment" matched the secondary-content names, and "media" or "widget" gave h1/h2 a negative cleaning weight. Real headings and whole docs sections were dropped. dev/0038 lost every "...Social Media Marketing..." heading, and dev/0662 (Flask docs) lost its "Cookies" section.

A survey of main found 77 headings (on 37 pages) that Boilerplate removed only because of an id made from their text: the WCXB reference keeps 67 of them and drops 10.

Change

  • New HeadingAnchors.mark, run once in the Preprocessor, marks an element whose id is made from its heading's text: the heading itself, a wiki headline span inside it, or a section whose first non-empty child is that heading. The comparison ignores case, accents, punctuation and separators, plus a one-letter prefix (h-, s-, also uppercase H-) and a trailing counter.
  • Dom.signature and Scorer.cleaning_weight leave a marked id out (Dom.name_id). Classes still count. Candidate scoring (Scorer.class_weight) keeps Readability's weight unchanged.
  • The id of a junk label is not marked, so the heading and the widget under it still go together, as they do today. A junk label is a heading that NOISE_HEADING or BOILERPLATE_TEXT matches ("Related posts", "Share this article", "Comments"), one whose whole text is a furniture or secondary-content name ("Feedback", "Table of contents", "Author bio"), or one that names reviews anywhere ("Customer reviews", "Ratings and Reviews"; reviews are only removed from product pages, where such a section is the reviews widget).
  • Only the label's words are matched. Heading-anchor markup adds a permalink to the heading text (Sphinx "¶", Hugo "#", a zero-width space in Docusaurus and VitePress), and the junk patterns are anchored at the end, so "Related posts¶" or "Comments¶" would otherwise count as content and keep their section.
  • WordPress widgets that keep their default title (<section id="recent-comments-2" class="widget widget_recent_comments"><h2>Recent Comments</h2>) are marked, so only their widget_* class names them. WordPress always adds that class, and they are still removed.

Results

WCXB, main -> this branch (F1; snippet rates and tokens where they moved):

type dev test
overall 0.8679 -> 0.8680 (+0.0001) 0.8918 -> 0.8919 (+0.0001)
article 0.9496 -> 0.9496 (+0.0001) 0.9503 -> 0.9504 (+0.0001)
collection 0.7218 -> 0.7218 0.8021 -> 0.8021
documentation 0.9319 -> 0.9327 (+0.0008) 0.9419 -> 0.9421 (+0.0003)
forum 0.7600 -> 0.7600 0.8191 -> 0.8191
listing 0.7280 -> 0.7280 0.7578 -> 0.7578
product 0.6789 -> 0.6789 0.8253 -> 0.8253
service 0.8381 -> 0.8379 (-0.0002) 0.8381 -> 0.8381
  • Snippet "with" rate: dev overall 0.814 -> 0.815, dev documentation 0.785 -> 0.791. The "without" rate did not change.
  • Tokens: dev overall 3073 -> 3075, dev documentation 5070 -> 5096. Test overall stayed at 2652.
  • 36 pages moved: 28 up, 8 down. The biggest gains were dev/2874 +0.060, test/5624 +0.030, dev/0662 +0.023, dev/0100 +0.020, and dev/0038 +0.015 (0.9833 -> 0.9982; all 9 of its headings are back).
  • Headings: 61 of the 67 wrongly removed headings come back. The other 6 count as junk labels under the rule, although the reference keeps them on their pages ("Table of Contents", "Related content", "Related Products", "Recommended Providers", "Navigation", "Adsense"). Of the 10 correct removals, 8 still go ("Feedback" x2, "Related posts" x3, "Related content", "Table of Contents", "Step Never prune text inside code blocks #7: Seek Recommendations and Reviews"). 2 come back, both content headings whose sections were already in the output: "Recommended Message Style", "How can I gather effective feedback on my API documentation?".
  • Across all rules, the output gains 81 heading lines on 34 pages (77 of them in the reference) and loses 3 on 2 pages (1 in the reference).
  • Empty sections: no added heading is left at the end of a page. One is followed directly by a same-level heading: dev/0676 "Social Auth" -> "Phone Auth". Main already shows "Phone Auth" that way, because link-cluster pruning removes both provider grids. The reference keeps both headings.
  • Scrapinghub 0.9451 -> 0.9451. Mozilla 0.9414 -> 0.9430.
  • Marking costs 0.24 ms per page on dev (0.7% of extraction time).
  • Synthetic checks against main: junk headings with a permalink ("Comments¶", "Related posts¶", "Share this article¶", "Related articles #", Docusaurus "Related posts" plus a zero-width space) and their sections are removed as on main. On a product page, "Ratings and Reviews" and "Ratings & Reviews" sections with anchor ids are removed as on main.

Regressions

  • dev/2850 (service) 0.7545 -> 0.7185: the "Why PostHog?" section is gone. Its wrapper's id why-posthog is the heading's anchor, and "post" inside "posthog" used to give it a +25 cleaning weight. Without that weight, conditional cleaning removes it. The anchor no longer counts either way.
  • dev/2871 (documentation) 0.7415 -> 0.7167: the Django section "Be careful with related_name and related_query_name" (s-be-careful-with-related-name...) is back. The reference leaves out that subsection and "Meta inheritance" but keeps the sections around them.
  • test/5634 -0.0031, test/5438 -0.0016, dev/2861 -0.0002: these headings are back, over content already in the output, but the reference leaves the heading text out.
  • dev/0109 -0.0008, dev/0071 -0.0004, dev/4431 -0.0001: the returned headings are in the reference. F1 drops by word counts only.
  • dev/0266 (service): the heading "Glowing Customer Reviews" (in the reference) stays removed, as on main, and F1 does not move. It names reviews, and the page declares a schema.org Product in its JSON-LD, so ProductReviews runs there and reads h-glowing-customer-reviews.
  • Not covered, synthetic only: sections whose anchored heading is a plural or a longer phrase of a noise name ("Sponsors", "Advertisements", "Newsletter Signup", "Join our newsletter", "Subscribe to our newsletter") are kept where main removed them by their id. Allowing plurals would also make "Cookies" junk, the Flask section this PR brings back, and "Sponsors" is content on a conference page. WCXB has no such section.

With #16

git merge-tree --write-tree origin/wcxb-losses HEAD reports one conflict, in spec/nokodown/boilerplate_spec.rb. Both PRs add a describe block in the same place, after "sections named like furniture". The resolution keeps both blocks: #16's "noise names" first, then "ids made from a heading's text". preprocessor.rb, scorer.rb and nokodown_spec.rb merge cleanly. The merge's lib/ is exactly #16 plus this PR's lib/ changes. #16's new rules use Dom.signature, so they skip anchors too. All 251 examples pass on the merge.

pr16 -> pr16 + this branch (local merge, not pushed):

type dev test
overall 0.8745 -> 0.8746 (+0.0001) 0.9002 -> 0.9003 (+0.0001)
article 0.9524 -> 0.9525 (+0.0001) 0.9538 -> 0.9539 (+0.0001)
collection 0.7292 -> 0.7292 0.8066 -> 0.8066
documentation 0.9321 -> 0.9329 (+0.0008) 0.9415 -> 0.9418 (+0.0003)
forum 0.7943 -> 0.7943 0.8559 -> 0.8559
listing 0.7321 -> 0.7321 0.7652 -> 0.7652
product 0.6865 -> 0.6865 0.8342 -> 0.8342
service 0.8475 -> 0.8473 (-0.0002) 0.8526 -> 0.8526

Scrapinghub 0.9468 -> 0.9468, Mozilla 0.9460 -> 0.9477. The same 36 pages move as on main, with the same single same-level empty heading (dev/0676).

How it was tested

  • bundle exec rspec: 238 examples, 0 failures. New synthetic specs:
    • heading_anchors_spec.rb: prefixes (also uppercase), counters, wiki ids, accents, apostrophes, a section opener after empty anchor spans, non-anchor ids, junk labels (also "Ratings and Reviews"), and junk labels with a "¶", a "#" before or after, or a zero-width space.
    • Boilerplate: anchored headings and sections are kept; junk labels (also a "Related articles¶" section), ids that are not anchors and furniture classes are still removed.
    • ProductReviews: a "Ratings and Reviews" section with an anchor id is removed.
    • End to end: an h2 whose anchor holds "media" survives header cleaning; a comments section headed "Comments¶" is still dropped.
    • Run against origin/main's lib/, only the 2 "keeps" specs fail (heading_anchors_spec.rb left out, since the module does not exist there). Run against this PR's first commit (a85f720), the 6 examples that cover the junk-label fix, the uppercase prefix and the reviews section fail.
  • WCXB dev and test via tmp/bench/eval.sh against main and against pr16. Scrapinghub and Mozilla via benchmark/quality.rb.
  • To find empty sections, I listed every heading the run adds over its base and checked what follows each one.

🤖 Generated with Claude Code

Alaa-abdulridha and others added 2 commits October 28, 2026 13:29
Sites build heading ids from the heading's text ("h-what-is-social-media",
"travel-related-coverage-2"), and docs generators put the same id on the
section the heading opens. The furniture, secondary and negative-weight
rules read words such as "social", "sponsor", "related" or "media" in those
ids and dropped real headings and sections. Such ids are now marked once in
the preprocessor and left out of signatures and cleaning weights, unless the
heading is itself a junk label ("Related posts", "Feedback").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Heading-anchor markup adds a permalink ("¶", "#", a zero-width space)
to the heading text, so "Related posts¶" or "Comments¶" slipped past the
end-anchored junk patterns and their sections were kept. Match the
patterns against the label's words only.

Also treat any label that names reviews as junk, so a product page's
"Ratings and Reviews" section is still removed, and accept an
uppercase one-letter prefix ("H-Social-Tips").

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Alaa-abdulridha
Alaa-abdulridha merged commit 91fa7c6 into main Nov 4, 2026
3 checks passed
@Alaa-abdulridha
Alaa-abdulridha deleted the heading-anchor-names branch November 4, 2026 17:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants